Environmental acoustic dosimetry with water event detection

ABSTRACT

In-ear sound pressure level, SPL, is determined that is caused by output audio being converted into sound by a headset worn by a user. The in-ear SPL is converted into a sound sample having units that are suitable for evaluating sound noise exposure. These operations are repeated to produce a sequence of sound samples during playback. This sequence of sound samples is then written to a secure database. Access to the database is authorized by the user. Other aspects are also described and claimed.

This non-provisional patent application claims the benefit of theearlier filing date of provisional application No. 62/855,956 filed Jun.1, 2019.

FIELD

An aspect of the disclosure here relates to digital signal processingtechniques for monitoring a user's environmental sound energy using awater event detector to improve accuracy.

BACKGROUND

Consumer electronic headsets have become increasingly popular withusers, because they reproduce media such as a music, podcasts and moviesound tracks with high fidelity while at the same time not disturbingothers who are nearby. While the listening experience with a headset isenjoyable, and the maximum sound output of a headset is limited inaccordance with hearing health safety standards, there is still a needto monitor the headset's sound output over relatively long periods oftime such as days and weeks, as part of personal hearing healthmonitoring that aims to avoid long term exposure to loud sounds.

BRIEF DESCRIPTION OF THE DRAWINGS

Several aspects of the disclosure here are illustrated by way of exampleand not by way of limitation in the figures of the accompanying drawingsin which like references indicate similar elements. It should be notedthat references to “an” or “one” aspect in this disclosure are notnecessarily to the same aspect, and they mean at least one. Also, in theinterest of conciseness and reducing the total number of figures, agiven figure may be used to illustrate the features of more than oneaspect of the disclosure, and not all elements in the figure may berequired for a given aspect.

FIG. 1 shows an example audio system having a headset and an audiosource device.

FIG. 2 is a flow diagram of a feedforward type acoustic dosimetryprocess for headset listening.

FIG. 3 shows an example volume curve determined for an unknown headset,which relates volume steps to maximum SPL output.

FIG. 4 is a flow diagram of an ambient environment acoustic dosimetryprocess.

FIG. 5 illustrates an aggregate dosimetry process that receives multipleinputs from different devices.

FIG. 6 is a diagram of an acoustic dosimetry process that uses a winddetector as a context input.

FIG. 7 is a diagram of an acoustic dosimetry process that uses a waterevent detector as a context input.

DETAILED DESCRIPTION

Several aspects of the disclosure with reference to the appendeddrawings are now explained. Whenever the shapes, relative positions andother aspects of the parts described are not explicitly defined, thescope of the invention is not limited only to the parts shown, which aremeant merely for the purpose of illustration. Also, while numerousdetails are set forth, it is understood that some aspects of thedisclosure may be practiced without these details. In other instances,well-known circuits, structures, and techniques have not been shown indetail so as not to obscure the understanding of this description.

FIG. 1 shows an example of an against-the-ear audio device 1 that ispart of an audio system in which an audio signal processing method foracoustic dosimetry of sound produce by the against-the-ear audio device1 can be implemented. The against-the-ear audio device 1 shown is anin-ear earbud (in-ear headphone which may be a sealing type that has aflexible tip, or it may be a non-sealing type also referred to as loosefitting.) The against-the-ear audio device 1 may be one of twoheadphones (left and right) that make up a headset. The methodsdescribed below can be implemented for one or both of the headphonesthat make up a headset. Alternatives (not shown) to the in-ear earbudinclude an on-the-ear headphone and an over-the-ear headphone. Theagainst-the-ear audio device 1 is shown in use, by its user (who mayalso be referred to as a listener or a wearer.) The against-the-earaudio device 1 has an against-the-ear acoustic transducer or speaker 2(arranged and configured to reproduce sound directly into an ear of itsuser), and may optionally have an external microphone 3 (arranged andconfigured to receive ambient sound directly) and an internal microphone4 (arranged and configured to directly receive the sound reproduced bythe speaker 2.) These microphones may be integrated in their respectiveaudio device housing. The microphones are not necessary for some of themethods described below (e.g., when estimating sound exposure due toheadphone playback.) The internal microphone 4 can be useful in makingan accurate measurement of in-ear sound pressure level, SPL. Also, theremay be more than one external microphone 3, such as a microphone arrayintegrated within a single housing of the against-the-ear audio device1, that produces multiple channels (multiple sequences of digital audioframes); any reference here to evaluating the strength of “a” microphonesignal is understood to be a more general reference to evaluating thestrength of one or more sound pickup channels, e.g., a single output ofa single microphone, a multi-channel output of a microphone array, or asingle output of a sound pickup beamforming process that receives themulti-channel output from the microphone array.

The methods described below are suitable for processing a digital outputaudio signal that is otherwise essentially ready to drive the speaker 2.The output audio is the result of an audio rendering process that maybegin with obtaining an input audio signal via a wired or wirelesscommunication interface, e.g., from a streaming music or podcast ormovie service over the Internet. The methods described below as well asthe audio rendering process and even the communication interface thatdelivers the input audio are performed by various electronic hardwarecomponents all of which may be integrated in a housing of theagainst-the-ear device 1. Thus, in instances where the housing of theagainst-the-ear device 1 has sufficient space and electrical power(e.g., from a rechargeable battery), all of the electronics that obtain,process and produce the transducer signal that drives the speaker 2 canbe placed in the same housing. The electronics may include an audioamplifier to drive the speaker 2 with the output audio signal, anoptional microphone sensing circuit or amplifier that receives theoptional microphone signals converts them into a desired format fordigital signal processing, and one or more digital processors (referredto here as “a processor”) and memory (e.g., solid state electronic datastorage circuitry) where the memory stores instructions for configuringthe processor (e.g., instructions to be executed by the processor) toperform the digital signal processing tasks discussed below in moredetail.

Note that it is also possible that some or essentially all of theelectronics referred to below as implementing the acoustic dosimetrytechniques reside in another device, separate from the against-the-eardevice 1. For instance, in the case of the against-ear-device 1 being aleft headphone or right headphone, the headphone may be connected to anaudio source device 5 shown in the example of FIG. 1 as a smartphone,via a wired connection (e.g., a computer bus connector that alsodelivers power, in which case there may be no need for a power source inthe headphone housing) or via a wireless connection (e.g., a BLUETOOTHlink.) In both cases, the output audio (that contains user content suchas a movie sound track, music or the voice of a far-end user during aphone call) has been rendered by one or more processors (“a processor”)in the audio source device 5, and much if not all of the processingdescribed below may also be performed by (or at) the processor in theaudio source device 5. For example, the instructions that configure orprogram the processor can be part of operating system software, or theycan be part of application software that is running on top of theoperating system program, in the audio source device 5. Thus, one ormore of the operations in the acoustic dosimetry process may beperformed by or at a processor of a smartphone, a smartwatch, a tabletcomputer, or other audio source device that can communicate with aheadset. For example, the headset may be a peripheral device of theaudio source device, or the headset and the audio device may becompanion devices, e.g., a headphone that can be paired with orconfigured to be used along with the audio source device and both can beconsidered personal devices for a particular user in the sense that theyare personally configured for a particular user. The audio source deviceprovides user content output audio, over a wired audio link or over awireless audio link to the headset, for playback through the speakers ofthe headset. Alternatively, one or more of the operations of theacoustic dosimetry process could be performed by processors in one orboth headphone housings of a headset or elsewhere in the headset(provided the headset has sufficient computing resources to do so.)

The acoustic dosimetry process measures or estimates in-ear SPL, e.g.,at or near an eardrum reference point, during user content headphoneplayback. In one aspect, the in-ear SPL is measured as follows. Thesignal from the internal microphone 4—see FIG. 1—which picks up sound inthe ear canal—may be processed into an equivalent SPL, using for examplelaboratory calibration results that include correction factors, e.g.,equalization, to be applied to the microphone signal. This conversion ofthe microphone signal into in-ear SPL may be performed during headphoneplayback (sound conversion through the headset worn by the user.)Alternatively, if the internal microphone 4 is not available, then theprocess flow in FIG. 2 may be used to estimate the in-ear SPL in a“feedforward” fashion.

Referring to FIG. 2, the process here begins with making laboratorymeasurements using several different ear simulators and manikins thatare wearing a manufactured headset, to produce sound output sensitivitydata for different types of ears and ear canals expected in the userpopulation. In one aspect, a nominal set of output sensitivity data isselected for an expected population of users. This factory calibrationprocedure may be performed on every manufactured headset, for example atproduction test, or it may be performed once for a group of similarheadsets. Statistical techniques may be used to find the best fit for agroup of similar headsets, and the best fit is then adopted for a largenumber of similar headsets. In any case, the calibration procedureassigns sound output sensitivity to each headset. The sound outputsensitivity gives a mathematical relationship between strength of theaudio signal that is driving the speaker 2 and the strength of theresulting sound output of the speaker 2 as worn by an expected user. Itmay vary as a function of auditory frequencies of interest, or it may beprovided as a statistical measure across an entire audible frequencyrange, e.g., as relationship between an average input and an averageoutput. It may be stored as a data structure in microelectronic memorywithin a housing of each headset. In one aspect, the against-the-earaudio device 1 can provide its stored, assigned output sensitivity data,that may include headphone acoustic output sensitivity and volume curveparameters, to an audio source device, for example as described below.

The against-the-ear audio device 1 may then be paired with the separate,audio source device 5 (e.g., as a wireless headset that pairs with asmartphone, a smartwatch, or a tablet computer through a wireless linksuch as a BLUETOOTH link, or as a wired headset that connects to theaudio source device over a wired link such as a serial peripheral buslike a Universal Serial Bus, USB, connection.) A processor in theheadset would then receive digital output audio, over the pairedconnection with the audio source device, and would then drive the audioamplifiers and speakers of the headset to reproduce as sound the outputaudio (also referred to as a playback signal that contains user contentfor example media such as music, podcast, video game, or a moviesoundtrack.)

The output sensitivity data may be transferred from the memory in theagainst-the-ear device 1 to memory in the audio source device 5, e.g.,over a BLUETOOTH link, where it is used by a processor executing anacoustic dosimetry algorithm or program for headset listening. Theprocess estimates in-ear sound pressure level, SPL, as follows. Stayingwith FIG. 2, the processor in the audio source device 5 computes ameasure of strength of the output audio that is being played back (block8.) This may be, for example as a root mean square, RMS value. Note thatthe output audio is a result of an audio rendering process that performsa conventional audio signal processing chain of operations upon an inputplayback signal (containing media such as music or a movie soundtrack.)These may include dynamic range adjustments, equalization, and gainadjustment for volume step. The processor then converts the RMS value ofsuch output audio into an in-ear SPL, by applying to the RMS value(multiplying it by) the received acoustic output sensitivity data (forthe presently used headset.) As an example, dB Full scale RMS values areconverted into in-ear SPL dB values. More generally, this is referred tohere as a conversion to in-ear SPL, based on output sensitivity (outputsensitivity conversion 10.)

Next, in block 11, the measure or estimate of in-ear SPL is converted tounits that are suitable for evaluating sound noise exposure (e.g., unitsspecified by a hearing health safety standard which may be a standard orcommonly defined metric for permissible sound noise exposure for hearinghealth.) For example, the in-ear SPL may be multiplied by a transferfunction (that has been determined in a laboratory setting) whichconverts in-ear SPL to an equivalent, free-field or diffuse fieldmeasurement of sound as would be picked up by an imaginary referencemicrophone that is located at some distance away from the user, asdefined by the hearing health safety standard. The transfer function maybe between a drum reference point, DRP, and a reference microphone. Theresult of the conversion is referred to here as a computed sound sample,for example in units of SPL dBA (A-weighted decibels).

The sound sample may be computed repeatedly over time, for example everysecond or other suitable interval during playback, and then accumulatedinto a batch or sequence of sound samples. The batch is then writteninto a database 12 of health data stored in memory of the audio sourcedevice 5. The database 12 may be secure in the sense that access to itneeds to be authorized by the individual user or owner of the healthdata. In addition to the sound samples, metadata such as the model ofthe headset that reproduced the output audio as sound, and theapplication program from which the output audio originated, may also bewritten as part of the batch. Note here that the sound samples writteninto the database cannot be used to recover identifying informationabout the media being played back, e.g., the title of a musical work orof a movie.

The stored or monitored sound samples may then be presented by anapplication program or app 14 (also being executed by the processor inthe audio source device 5) for visualization on a graphical userinterface of the audio source device. For example, and still referringto FIG. 2, a health application program as an example of the app 14 maybe given authorization to access the locally stored health database 12to retrieve the sound samples, and computes various statistical measuresof the collected sound samples, such as Leq dBA (average) over certaintime intervals. The health app may then “show” the user their soundexposure that is due to playback by the against-the-ear audio device 1.The health app may also visualize to the user which portions of thesound samples were produced by which apps (e.g. a music app, a videogame app, and a movie player), and which models of against the ear audiodevices produced which sound samples. It is expected that the user mayuse several different models of headsets for listening, such as in-earwired earbuds, in-ear wireless earbuds, and on-the-ear headphones, atdifferent volume steps or with different media. This useful informationmay be monitored and reported to the user by the health app. Other waysof reporting useful information to the user about such collected soundsamples (acoustic dosimetry) are possible.

In the case where the factory calibration output sensitivity informationis not available, the processor in the audio source device 5 can stilldetect that an unknown against-the-ear audio device is being used toconvert the output audio into sound (e.g., detect that a computer busconnector has been plugged into the audio source device, or a BLUETOOTHlink has been established with a headset.) The processor uses thatinformation to select a volume curve. As seen in FIG. 3, a volume curvemay be selected, that best fits the characteristics of a number ofdifferent types of commercially available headsets (and their publishedsound output characteristics.) These headsets may have been tested tomeasure their sound output (e.g., in-ear SPL) versus volume step andwhile a nominal audio signal processing chain is applied (for examplespecific to the particular app that is producing the present, outputaudio in the audio source device). The present volume step (at which thepresent output audio is being produced) is then applied to the selectedvolume curve, to obtain an estimate of the maximum SPL at that volumestep that is assigned to the connected, unknown headset. A derivedoutput sensitivity relationship between RMS and in-ear SPL is thendetermined for the unknown headset based on the maximum SPL given by theselected volume curve. The derived output sensitivity relationship isthen inserted into the process flow described above and shown in FIG. 2,to compute sound samples in units of SPL dBA (or other units that aresuitable for evaluating sound exposure such as per a standard orcommonly defined hearing health sound noise exposure calculations.)

It should be noted that the in-ear SPL measurements or estimates asobtained above should be adjusted in cases where there is sound energybeing produced by the speaker 2 that comes from other than the playbacksignal. For instance, the headset may have acoustic noise cancellation,ANC, capability, ambient sound transparency capability (where ambientsound is picked up and actively reproduced during playback, to give thewearer a sense of awareness of her environment), or ambient soundenhancement (where the ambient sound is picked up and amplified whenbeing reproduced so as to compensate for hearing loss or meet a hearingpreference of the wearer.) In such cases, the headset may electronicallyidentify such capability during initial pairing with the audio sourcedevice, in response to which the acoustic dosimetry process may beconfigured to adjust its in-ear SPL measurements or estimatesaccordingly to take into account any such added sound energy.Alternatively, since such additional sound energy is picked up to someextent by the internal microphone 4, it will thus be inherentlyaccounted for in the in-ear SPL measurement made using the signal fromthe internal microphone 4.

Turning now to FIG. 4, this figure illustrates an example of how toperform acoustic dosimetry of sound that is in the ambient environmentof a user (also referred to as environmental sound or noise exposure),e.g., sound from loudspeakers, vehicle sounds, and sound from any othersources in the ambient environment. The external microphone 3 shown inthe example of FIG. 1 as being embedded on the outer face of a housingof a headphone and which can as a result perform direct sound pickup inthe ambient environment, is now embedded in or integrated in a housingof a smartwatch 16 that is worn around the wrist of the user as shown.The smartwatch 16 may be paired for wireless digital communications withthe audio source device 5 (e.g., a smartphone or a tablet computer ofthe user.) Computations similar to those described above in FIG. 2 thatare performed upon the output audio to determine its strength (e.g.,RMS) are now performed upon a microphone signal that is generated by anaudio device worn by the user. For example, FIG. 4 shows this signal asbeing generated by the external microphone 3. Note here that there maybe more than one external microphone 3, such as a microphone arrayintegrated within a single housing of the audio source device 5 or ofthe smartwatch 16, that produces multiple channels as multiple sequencesof digital audio frames; any reference here to evaluating the strengthof “a” microphone signal is understood to be a more general reference toevaluating the strength of one or more sound pickup channels, e.g., asingle output of a single microphone, a multi-channel output of amicrophone array, or a single output of a sound pickup beamformingprocess that receives the multi-channel output from the microphonearray.

A desired result of ambient environment acoustic dosimetry is to computea sound sample of the ambient environment, e.g. in units of SPL dBA, byconverting the strength of the microphone signal into suitable units forexample as defined by a health hearing safety standard. In one aspect,those computations are performed by a low power or auxiliary processorthat is integrated in a housing of the smartwatch 16, which helps reducepower consumption (an important goal for the smaller, lower energydensity batteries that power the smart watch.) An additional benefithere of having the sound sample computations performed by the low poweror auxiliary processor (which also serves to detect a voice triggerphrase in the signal from the external microphone 3) is that theexternal microphone signal is only buffered for a short time interval,e.g., around one second, and as such does not capture any privateconversations that may be present in the ambient environment. The lowpower or auxiliary processor is one that is continually processing theexternal microphone signal to detect a voice trigger, while a high poweror primary processor in the smart watch is in a low power state (e.g.,sleep state.) Once the voice trigger is detected, the high power orprimary processor in the smartwatch is transitioned to a high powerstate (e.g., awake state) in which it can execute the more powerintensive tasks of the virtual assistant program.

The process flow of FIG. 4 for ambient/environmental acoustic dosimetryhas some differences as compared to the headset acoustic dosimetry orheadphone audio exposure given in FIG. 2. These include replacing theuser content playback signal with the external microphone signal,replacing the output sensitivity conversion 10 with an input sensitivityconversion 18 (e.g., one that relates dB full scale of the signalproduced by the external microphone 3 to SPL that is incident on theexternal microphone 3) and modifying block 11 to use a free-field ordiffuse field transfer function that is between the external microphone3, instead of the DRP, and the reference microphone of the hearinghealth safety standard which is depicted in dotted lines in FIG. 4.

The aspects depicted in FIG. 4 also include storage of the healthdatabase 12 to which the sound samples are written in batches asdescribed above, within memory of the smartwatch 16 rather than withinmemory of the audio source device 5. In addition, a visualizationapplication 17 is also being executed by the high power or primaryprocessor in the smartwatch 16, which produces visuals to be displayedon a screen of the smartwatch to the user, showing the levels of ambientsound exposure and their respective durations. The visualizationapplication 17 may also generate a visual or haptic notification 19 inthe smartwatch 16, when it determines that the collected sound samplesare above a loudness threshold over a time interval threshold (too loud,for too long), which thresholds may be defined by a hearing healthsafety standard. The notification 19 informs the user that their ambientenvironment may be too loud for hearing health, and is also referred tohere as a “loud environment” notification.

Another aspect of the disclosure here is an “aggregate” acousticdosimetry process, that may be performed by a programmed digitalprocessor, which estimates the total sound or hearing exposure (“totalsound exposure”) for a particular user. Computing the total soundexposure is a holistic view in the sense that the processor collectssound measurement inputs from microphones that are in multiple audiodevices, respectively, any one of which may be for example within aradius of two meters from the user. Each of the audio devices has aseparate housing (containing its respective one or more microphones) andis a different type of device, such as a desktop computer, a laptopcomputer, a tablet computer, a smartphone, a smartwatch, and a headset.The audio devices may be associated with a particular user, in the senseof how a smartphone, a smartwatch, and a headset are typically describedas being “owned by” a particular user (e.g., configured to be personalto the user.) One or more operations of the aggregate acoustic dosimetryprocess may be performed by a processor of a smartphone, a smartwatch, atablet computer, or other auto source device that could communicate withthe headset. For example, the headset may be a peripheral device of theaudio source device, or the headset and the audio device may becompanion devices, e.g., a headphone that can be paired with orconfigured to be used along with the audio source device and both can beconsidered personal devices for a particular user in the sense that theyare personally configured for a particular user. The audio source deviceprovides user content output audio, over a wired audio link or over awireless audio link to the headset, for playback through the speakers ofthe headset. Alternatively, one or more of those operations could beperformed by processors in one or both headphone housings of a headsetor elsewhere in the headset (provided the headset has sufficientcomputing resources to do so and is able to communicate with otherdevices of the user to obtain the inputs that are needed by theaggregate dosimetry process.) The aggregate process may interpret thesound measurements using other inputs, referred to here as contextinputs which are sources of information that refer to or shed light onthe situation in which the user finds themselves (during a given soundmeasurement.) The total sound exposure is then stored as a time sequenceof in-ear SPL values (e.g., in a secure database access to which isauthorized by the particular user.) The total sound exposure is thusindicative of the actual sound level to which the user's ear drum hasbeen exposed over a relatively long period of time such as hours, daysor weeks.

As seen in FIG. 5 the sound measurement inputs to the aggregatedosimetry process may include two or more of the following that occur atdifferent times while the user is exposed to various sound levels:estimates or measurements of headphone playback sound; soundmeasurements of environmental noise made by an external microphone 3that is integrated in the user's smartwatch 16; sound measurements madeby an internal microphone 4 that is integrated in an against-ear-audiodevice 1 such as a headphone that is worn by the user and that mayrepresent either i) headphone playback sound or ii) environmental soundnoise that has leaked past the headphone into the user ear, or both;sound measurements of environmental noise made by an external microphone3 that is integrated in the against-the-ear audio device 1;environmental noise measurements made by an external microphone 3 thatis integrated in a smartphone or in another audio source device 5belonging to the user, for example during a phone call; and soundmeasurements of environmental noise made by an external microphone 3 ofan audio device such as a desktop computer 19 or a lap top computer or atablet computer. The processor may then convert these diverse soundmeasurement inputs into a common format, such as in-ear SPL or a hearinghealth safety standard (see above for example in FIG. 2 and FIG. 4.) Theconverted or common format SPL values are stored as a time sequence ofsound samples, also referred to here as an aggregate dosimetry timesequence that reflects the actual in-ear SPL over time.

As part of the aggregate dosimetry process, the processor may in somecases perform an algorithm that determines which of two or morecontemporaneously arriving inputs (e.g., time stamped within the sametime interval, or received contemporaneously), will better or moreaccurately reflect the actual in-ear SPL. In one instance, the inputwith the highest confidence level is selected while the othercontemporaneous inputs are ignored. In other instances, one of the soundmeasurement inputs is weighted more than the others, before beingcombined into a single input value (e.g., an estimated in-ear SPL.) Forinstance, consider the case where a headphone playback estimate, that isbased on the current user settable volume at which certain user contentsuch as a movie soundtrack or music is being played back throughheadphones, is received contemporaneously with an ambient noisemeasurement; the user in that case is being exposed to various types ofsounds at the same time, as they are both listening to music playback ontheir headphones and are also in a loud ambient environment that couldbe heard despite the passive attenuation by the worn headset.

The processor stores a time sequence of such estimated, in-ear SPLvalues that reflects the user's hearing exposure as it varies over time,for example over a period of hours, days or weeks. This time sequencemay be analyzed by the processor to determine when the in-ear SPLexceeds a given level threshold over a given time interval, in responseto which the processor signals that the “too loud” notification 19 beimmediately generated and presented to the user. In general, too loudnotifications 14 may be generated in one or more of the user's devices,in response to the estimated in-ear SPL exceeding level thresholdsduring short term time frames or intervals (e.g., a few seconds) as wellas during long term time frames (e.g., an hour, a few hours, a day, afew days, a week.) The too loud notification 14 may for example includea haptic alert on the user's smartwatch 16 or on their smartphone orother audio source device 5. The too loud notification 14 may include inaddition to the haptic alert a contemporaneous voice alert (e.g.,“consider reducing the volume by two clicks”) or an audible alert (e.g.,an alarm sound) that may be played back through a speaker of the user'sheadphone, smartphone, tablet computer, or laptop computer. The too loudnotification 14 may also include a contemporaneous visible alert such asa pop-up on a display screen of any one or more of the user's devices.

The time sequence of sound samples (e.g., estimated in-ear SPL values)generated by the aggregate dosimetry process may be tagged (by theprocessor) with metadata that specifies whether a particular timeinterval of sound samples is primarily due to headphone playback orprimarily due to ambient noise. That time interval and its metadata maybe interpreted by the dosimetry process to determine that the user'sambient or environmental noise is exceeding a given level threshold anda given duration threshold, in that particular time interval.Alternatively, the sequence and metadata may be interpreted by theprocessor to determine that headphone playback is exceeding a givenlevel threshold and a given duration threshold, over a particular timeinterval. In response, too loud notifications may be signaled in eachinstance.

The time sequence of sound samples may cover a period of hours, days, orweeks. Location information (as defined by for example globalpositioning system, GPS, coordinates or other means for defining ageographic location or a place where the user is located) may beassociated with or tag the time sequence. When tagged with locationinformation, the time sequence of sound samples may be used by theprocessor to produce a map of what it determines to be loud locations(e.g., where it has determined that the ambient noise levels and theirtime durations exceed a hearing health standard.)

In another aspect of the aggregate dosimetry process, the processordetermines or logs information on why or how did a loud soundmeasurement (the in-ear SPL exceeded level and duration thresholds)happen. Such context information may be received as one more contextinputs to the process (see FIG. 5.) For example, referring now to FIG.6, one of the context inputs may be an output of a wind detector (a winddetection algorithm being performed by a digital processor.) In block36, the wind detector processes each digital audio frame (e.g., 10 msecto 50 msec long) of a received sequence of digital audio frames of asound pickup channel (e.g., an output of a single microphone, or anoutput of a sound pickup beamforming process that receives amulti-channel output from a microphone array) to generate an indicationsuch as a confidence level, a binary yes/no, or an estimated wind speed,as to whether a given frequency component in the particular audio frameis due to wind (rather than some ambient sound source.) This contextinput from the wind detector may then be used by the aggregate dosimetryprocess to decide, in block 39, whether to log or reject, oralternatively reduce the weight of, a contemporaneously received soundmeasurement in block 37, as being due to wind (rather than due to a loudsound produced by a sound source in the ambient environment.)

To further ensure accuracy of the computed, total sound exposure, theaggregate acoustic dosimetry process may determine whether or not theuser has active and passive hearing protections in place during thesound measurements, and on that basis applies an attenuation orreduction to the sound level measurements. Active hearing protection maybe acoustic noise cancellation, ANC, in which a sound transducer (e.g.,an earpiece speaker in a headset, a loudspeaker) is being driven toproduce anti-noise that is cancelling the ambient noise that wouldotherwise be heard by the user.

Passive hearing protection may include ear plugs 20 (see FIG. 5), andblockage of the ear due to the against the ear audio device 1 such as aheadphone, or due to a smartphone being held against the user's ear, anda result attenuating the ambient noise due to a partial or completeacoustic seal against the ear. In some cases, the processor may havepredetermined, stored knowledge of how much to deduct from soundmeasurements, based on knowledge of the specific type of passive hearingprotection being used in a given instance (e.g., sealing type earbuds,loose fitting earbuds, and handset or against the ear usage of asmartphone all of which may be provided by the same entity that is alsoproviding the aggregate dosimetry process.) In other cases, the user maybe wearing third party hearing protection whose attenuation effects areunknown to the user and unknown to the provider of the acousticdosimetry process.

Part of the aggregate dosimetry process may be a process for trackingover time the usage (by a particular user) of passive hearingprotection, such as whether or not the user is using passive hearingprotection at a concert, and the type of hearing protection such a pairof headphones or a pair of dedicated ear plugs, and what level ofattenuation the hearing protection is designed to provide, and loggingsuch usage ever time. This results in a time sequence of attenuationvalues that can be applied to reduce a synchronized time sequence ofenvironmental and headphone sound exposure measurements, to result inmore accurate “net” sound exposure samples. In one instance, the processcould prompt the user to manually select from a list of most popularhearing protections, including third party ones that are different thanthose from the provider of aggregate dosimetry process, the one thatthey are currently wearing.

A further difficulty in determining the correct amount of passiveattenuation is uncertainty in how the hearing protection is physicallyfitting with the user's ear. For example, the condition of the fit of aheadphone or a dedicated ear plug against the user's ear stronglyimpacts the amount of passive attenuation. When deciding on such anattenuation value, care should be taken to avoid underestimating thesound dose that the user is being exposed to.

Acoustic Dosimetry Based on Processing Touch Images for Rejecting WaterInduced Apparent Touches

Another one of the inputs to the aggregate dosimetry process in FIG. 5may be an output of a water event detector (a water detection algorithmbeing performed by a digital processor.) The water event detectoroutputs an indication of the presence of water on a particular audiodevice of the user. That audio device may be, for example, the user'ssmartwatch that is being worn by the user while the user is in a showeror outside where it is raining, or the user's smartphone or otherportable consumer electronics device that is likely to encounter waterin the ambient environment. This serves to provide the dosimetry processcontext for when not to accept, as a valid sample, an ambient soundlevel measurement made by that audio device. That is because an externalacoustic port in the device is likely being struck with water or iscontinuously covered with water and, as a result, a microphone that is“listening” through that port is likely to generate an erroneous soundlevel measurement at that point in time.

Referring now to FIG. 7, this is a flow diagram of a method for acousticdosimetry in which a water event detector provides a context input. Themethod may be performed by a digital processor and a touch sensorsubsystem within an audio device such as a smartwatch or a smartphone. Adigital touch image is acquired from a touch sensitive surface of atouch sensor subsystem (e.g., a touch screen in the device), and isdigitally processed for rejecting apparent touches caused by water(block 35.) An ambient sound level measurement is made using amicrophone in the device (block 37.) A determination is made as towhether or not the ambient sound level measurement should be logged as avalid sound exposure sample for acoustic dosimetry, or if its weightingshould be reduced for when it is combined with other sound measurementsinputs (block 38.) This is done based on a result of having processedthe touch image, e.g., how much if any of the touch image is determinedto be caused by water. In other words, the touch sensing surface in thedevice is being used to detect water events that become one context forwhen not to log a measurement of ambient sound level (when using an“open to the ambient” microphone within the audio device.)

In one aspect, processing the touch image includes discriminatingbetween i) floating (or electrically ungrounded) water on the touchsensitive surface and ii) a finger touch (which is electricallygrounded.) In another aspect, processing the touch image includesdetermining there is water on the touch sensitive surface andclassifying the determined water on the touch sensitive surface asstatic (e.g., one or more droplets) or dynamic (e.g., water is flowinglike when the device is worn by the user while taking a shower.) In thataspect, the dosimetry process does not log the ambient sound levelmeasurement if the water is classified as dynamic, and does log it ifthe water is classified as static. In other words, the ambient soundlevel measurement is logged if the water is classified as static, butnot if the water is classified as dynamic.

In yet another aspect, processing the touch includes determining anamount of water on the touch sensitive surface, by for example theprocessor counting a number of pixels in the touch image. The ambientsound level measurement is not logged if the determined amount of wateris greater than a threshold, but is logged if it is less than thethreshold. The dosimetry process may then send a loud environmentnotification to its user, if the logged ambient sound level measurementsremain above a threshold for at least a threshold amount of time.

The touch sensitive surface and the microphone may be integrated into ahousing of a portable consumer electronics device. In the case where thewater event detection process is being performed in a smartwatch (wherethe touch sensitive surface and perhaps also the microphone isintegrated in or part of the smartwatch), a determination of whether ornot an ambient sound level measurement is logged is further based on theprocessor having detected whether or not the smartwatch is on a user'swrist (e.g., by executing an on-wrist detection algorithm.)

A method for acoustic dosimetry in which water events are detected by aprocessor, and then used as a context input to decide whether or not tolog contemporaneous loud sound events, may proceed as follows. A timesequence of touch images is acquired from a touch sensitive surface, andis processed in order to detect floating water on the touch sensitivesurface, as a time sequence of water events. Ambient sound levelmeasurements are made using a microphone, as a time sequence, and thoseare processed to detect loud sounds as a time sequence of loud events.In time intervals over which the time sequence of water events does notcorrelate with the time sequence of loud events, the loud events arelogged for acoustic dosimetry. But in time intervals over which thewater events correlate with the loud events, and the water eventsindicate an amount of floating water that exceeds a threshold or thatthere is dynamic water (water is flowing across the touch sensitivesurface), the loud events are not logged (or they disregarded forpurposes of acoustic dosimetry.) For instance, for a time intervalduring which the water events correlate with contemporaneous loudevents, the processor foregoes the presentation of a user notificationthat indicates exposure to loud sound even though the loud events exceeda level threshold for at least a threshold amount of time. Such waterevents could indicate an amount of floating water that exceeds athreshold, or they could indicate that there is dynamic water on thetouch sensitive surface.

The acoustic dosimetry process may be continuously monitoring the timesequence of logged loud events, and where in response to the logged loudevents exceeding a level threshold for at least a threshold amount oftime the processor will signal a device of the user, e.g., the user'ssmartwatch or the user's smartphone, to present a user notification thatindicates exposure to loud sound. As suggested above, the usernotification may include the display of text or graphics by the touchsensing surface.

Revisiting FIG. 5, the aggregate acoustic dosimetry process is fed avariety inputs some of which are direct sound level measurements whileothers provide the process with context as why or how a particularlyloud sound event is taking place. One such context input discussed aboveis an output of a wind detector. Another context input that was alsodiscussed above and that may be interpreted to decide not to log anambient sound level measurement or reduce its weight when combined withother sound level estimates is an output of a water event detector.There may be other context inputs as well, such as the following: adetermination that a smartwatch is not on a user's wrist; adetermination that a smartwatch environment is windy; and adetermination of user context based on accelerometer/gyroscope/magneticsensor (compass) signals in a smartwatch or in a smartphone, e.g., thatthe user is walking, running, riding a bicycle, riding in or driving acar. The context inputs may be interpreted by the processor to forexample determine that a microphone which is being used to make soundmeasurements is experiencing high air flow which affects the accuracy ofthe sound measurement. In addition to sensor-originated context inputs,there may also be manually provided context inputs such as a response bythe user to a voice prompt or visual prompt that is asking whether ornot the user is in water.

An article of manufacture for headset playback acoustic dosimetry, suchas a microelectronic integrated circuit device, may comprise memoryhaving stored therein instructions that configure a processor of anaudio source device to perform any one or more of the operationsdescribed above.

A digital signal processing method for headset playback acousticdosimetry at a processor of an audio source device, the methodcomprising: a) determining in-ear sound pressure level, SPL, that iscaused by output audio being converted into sound by a headset worn by auser, wherein the headset is a peripheral device of the audio sourcedevice; b) converting the in-ear SPL into a sound sample having unitsfor sound noise exposure; repeating a) and b) a plurality of times toproduce a time sequence of sound samples; and writing the time sequenceof sound samples to a secure database access to which is authorized bythe user. Determining in-ear SPL comprises measuring in-ear SPL from anaudio signal produced by an internal microphone of the headset.Determining in-ear SPL comprises: receiving from the headset previouslydetermined acoustic output sensitivity data that is stored in theheadset; determining strength of the output audio; and multiplying theacoustic output sensitivity data with the strength of the output audio.The method further comprising writing metadata for the time sequence ofsound samples to the database wherein the metadata identifies a model ofthe headset and an application program from which the output audiooriginated. The method further comprising executing by the processor inthe audio source device an application program that accesses the timesequence of sound samples in the database and computes a statisticalmeasure of the time sequence of sound samples and visualizes thestatistical measure through a graphical user interface of the audiosource device. The method wherein determining in-ear SPL comprisesdetecting that the headset is connected to the audio source device butthat the output sensitivity data for the connected headset is notavailable, and in response selecting a volume curve; applying a presentvolume step to the volume curve to obtain a maximum in-ear SPL; andderiving an output sensitivity relationship based on the maximum in-earSPL given by the volume curve. The method wherein determining the in-earSPL accounts for acoustic noise cancelation, ambient sound transparency,or ambient sound enhancement by the headset.

An audio source device with headset playback acoustic dosimetrycapability, the audio source device comprising: a processor; and memoryhaving stored therein instructions that configure the processor to a)determine in-ear sound pressure level, SPL, that is caused by outputaudio being converted into sound by a headset worn by a user, whereinthe headset is a peripheral device of the audio source device, b)convert the in-ear SPL into a sound sample having units for sound noiseexposure, repeat a) and b) a plurality of times to produce a timesequence of sound samples and write the time sequence of sound samplesto a secure database access to which is authorized by the user. Theaudio source device wherein the processor determines in-ear SPL bymeasuring in-ear SPL from an audio signal produced by an internalmicrophone of the headset. The audio source device wherein the processordetermines in-ear SPL by: receiving from the headset previouslydetermined acoustic output sensitivity data that is stored in theheadset; determining strength of the output audio; and multiplying theacoustic output sensitivity data with the strength of the output audio.The audio source device wherein the memory has further instructions thatconfigure the processor to write metadata for the time sequence of soundsamples to the database wherein the metadata identifies a model of theheadset and an application program from which the output audiooriginated. The audio source device wherein the memory has furtherinstructions that configure the processor to execute an applicationprogram that accesses the time sequence of sound samples in the databaseand computes a statistical measure of the time sequence of sound samplesand visualizes the statistical measure through a graphical userinterface of the audio source device. The audio source device whereinthe processor determines in-ear SPL by: detecting that the headset isconnected to the audio source device but that the output sensitivitydata for the connected headset is not available, and in responseselecting a volume curve; applying a present volume step to the volumecurve to obtain a maximum in-ear SPL; and deriving an output sensitivityrelationship based on the maximum in-ear SPL given by the volume curve.The audio source device wherein the processor when determining thein-ear SPL accounts for acoustic noise cancelation, ambient soundtransparency, or ambient sound enhancement by the headset.

A smartwatch comprising: a smartwatch housing having integrated thereina touch sensitive surface, a microphone, a processor, and memory havingstored therein instructions that configure the processor to perform adigital signal processing method for ambient sound acoustic dosimetry inwhich the processor acquires a touch image from the touch sensitivesurface, and processes the touch image for rejecting apparent touchescaused by water, makes an ambient sound level measurement using themicrophone, contemporaneous with acquiring the touch image; anddetermines whether or not the ambient sound level measurement is loggedas a valid sound exposure sample for acoustic dosimetry, based onprocessing the touch image. The smartwatch wherein the processorprocesses the touch image by discriminating between i) floating water onthe touch sensitive surface and ii) a finger touch. The smartwatchwherein the processor processes the touch image by: determining there iswater on the touch sensitive surface; and classifying the determinedwater on the touch sensitive surface as static or dynamic. In oneaspect, the ambient sound level measurement is not logged if the wateris classified as dynamic, and is logged if the water is classified asstatic. In another aspect, the ambient sound level measurement is loggedif the water is classified as static, but not if the water is classifiedas dynamic. The smartwatch of claim wherein the processor processes thetouch image by determining an amount of water on the touch sensitivesurface. The smartwatch wherein the processor determines an amount ofwater by counting a number of pixels in the touch image. The ambientsound level measurement is not logged if the determined amount of wateris greater than a threshold. The smartwatch wherein the memory hasfurther instructions that configure the processor to log the ambientsound level measurement if the determined amount of water is less than athreshold.

As described above, one aspect of the present technology is thegathering and use of data available from specific and legitimate sourcesto monitor sound exposure when using headphones. The present disclosurecontemplates that in some instances, this gathered data may includepersonal information data that uniquely identifies or can be used toidentify a specific person. Such personal information data can includedemographic data, location-based data, online identifiers, telephonenumbers, email addresses, home addresses, data or records relating to auser's health or level of hearing impairment, date of birth, or anyother personal information.

The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used tomonitor long term exposure to sound to promote hearing health inaccordance with their preferences.

The present disclosure contemplates that those entities responsible forthe collection, analysis, disclosure, transfer, storage, or other use ofsuch personal information data will comply with well-established privacypolicies and/or privacy practices. In particular, such entities would beexpected to implement and consistently apply privacy practices that aregenerally recognized as meeting or exceeding industry or governmentalrequirements for maintaining the privacy of users. Such informationregarding the use of personal data should be prominent and easilyaccessible by users, and should be updated as the collection and/or useof data changes. Personal information from users should be collected forlegitimate uses only. Further, such collection/sharing should occur onlyafter receiving the consent of the users or other legitimate basisspecified in applicable law. Additionally, such entities should considertaking any needed steps for safeguarding and securing access to suchpersonal information data and ensuring that others with access to thepersonal information data adhere to their privacy policies andprocedures. Further, such entities can subject themselves to evaluationby third parties to certify their adherence to widely accepted privacypolicies and practices. In addition, policies and practices should beadapted for the particular types of personal information data beingcollected and/or accessed and adapted to applicable laws and standards,including jurisdiction-specific considerations that may serve to imposea higher standard. For instance, in the US, collection of or access tocertain health data may be governed by federal and/or state laws, suchas the Health Insurance Portability and Accountability Act (HIPAA);whereas health data in other countries may be subject to otherregulations and policies and should be handled accordingly.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, the presenttechnology can be configured to allow users to select to “opt in” or“opt out” of the sound exposure processing.

Moreover, it is the intent of the present disclosure that personalinformation data should be managed and handled in a way to minimizerisks of unintentional or unauthorized access or use. Risk can beminimized by limiting the collection of data and deleting data once itis no longer needed. In addition, and when applicable, including incertain health related applications, data de-identification can be usedto protect a user's privacy. De-identification may be facilitated, whenappropriate, by removing identifiers, controlling the amount orspecificity of data stored (e.g., collecting location data at city levelrather than at an address level), controlling how data is stored (e.g.,aggregating data across users), and/or other methods such asdifferential privacy.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data.

To aid the Patent Office and any readers of any patent issued on thisapplication in interpreting the claims appended hereto, applicant wishesto note that they do not intend any of the appended claims or claimelements to invoke 35 U.S.C. 112(f) unless the words “means for” or“step for” are explicitly used in the particular claim.

While certain aspects have been described and shown in the accompanyingdrawings, it is to be understood that such are merely illustrative ofand not restrictive on the broad invention, and that the invention isnot limited to the specific constructions and arrangements shown anddescribed, since various other modifications may occur to those ofordinary skill in the art. The description is thus to be regarded asillustrative instead of limiting.

What is claimed is:
 1. A method for acoustic dosimetry, comprising:acquiring a touch image from a touch sensitive surface, and processingthe touch image for rejecting apparent touches caused by water; makingan ambient sound level measurement using a microphone, contemporaneouswith acquiring the touch image; determining whether or not the ambientsound level measurement is logged as a valid sound exposure sample foracoustic dosimetry, based on processing the touch image; and sending aloud environment notification to a user, if logged ambient sound levelmeasurements remain above a threshold for at least a threshold amount oftime.
 2. The method of claim 1 wherein processing the touch imagecomprises discriminating between i) floating water on the touchsensitive surface and ii) a finger touch.
 3. The method of claim 1wherein processing the touch image comprises: determining there is wateron the touch sensitive surface; and classifying the determined water onthe touch sensitive surface as static or dynamic.
 4. The method of claim3 wherein the ambient sound level measurement is not logged if the wateris classified as dynamic, and is logged if the water is classified asstatic.
 5. The method of claim 3 further comprising logging the ambientsound level measurement if the water is classified as static, but not ifthe water is classified as dynamic.
 6. The method of claim 1 whereinprocessing the touch image comprises determining an amount of water onthe touch sensitive surface.
 7. The method of claim 6 whereindetermining an amount of water comprises counting a number of pixels inthe touch image.
 8. The method of claim 6 wherein the ambient soundlevel measurement is not logged if the determined amount of water isgreater than a threshold.
 9. The method of claim 6 further comprisinglogging the ambient sound level measurement if the determined amount ofwater is less than a threshold.
 10. The method of claim 1 wherein thetouch sensitive surface is part of a smartwatch, and wherein determiningwhether or not the ambient sound level measurement is logged is furtherbased on detecting whether or not the smartwatch is on a user's wrist.11. A smartwatch comprising: a smartwatch housing having integratedtherein a touch sensitive surface, a microphone, a processor, and memoryhaving stored therein instructions that configure the processor toperform a digital signal processing method for ambient sound acousticdosimetry in which the processor acquires a touch image from the touchsensitive surface, and processes the touch image for rejecting apparenttouches caused by water, makes an ambient sound level measurement usingthe microphone, contemporaneous with acquiring the touch image; andlogging the ambient sound level measurement as a valid sound exposuresample for acoustic dosimetry, based on processing the touch image forrejecting apparent touches caused by water.
 12. The smartwatch of claim11 wherein the processor processes the touch image by discriminatingbetween i) floating water on the touch sensitive surface and ii) afinger touch.
 13. The smartwatch of claim 11 wherein the processorprocesses the touch image by: determining there is water on the touchsensitive surface; and classifying the determined water on the touchsensitive surface as static or dynamic.
 14. The smartwatch of claim 13wherein the ambient sound level measurement is not logged if the wateris classified as dynamic, and is logged if the water is classified asstatic.
 15. The smartwatch of claim 13 wherein the memory has furtherinstructions that configure the processor to log the ambient sound levelmeasurement if the water is classified as static, but not if the wateris classified as dynamic.
 16. The smartwatch of claim 11 wherein theprocessor processes the touch image by determining an amount of water onthe touch sensitive surface.
 17. The smartwatch of claim 16 wherein theprocessor determines an amount of water by counting a number of pixelsin the touch image.
 18. The smartwatch of claim 16 wherein the ambientsound level measurement is not logged if the determined amount of wateris greater than a threshold.
 19. The smartwatch of claim 16 wherein thememory has further instructions that configure the processor to log theambient sound level measurement if the determined amount of water isless than a threshold.
 20. The smartwatch of claim 11 wherein the memoryhas further instructions that configure the processor to log the ambientsound level measurement based on detecting the smartwatch is on a user'swrist.